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Evaluation of the Effectiveness of Online Resources in Developing 
Student Critical Thinking: Review of Literature and Case Study of a 
Critical Thinking Online Site 


A graduate's ability to be a critical thinker is expected by many employers; therefore development of students’ 
critical-thinking skills in higher education is important. There is also a perception that today’s students are 
technologically "savvy", and appreciate the inclusion of a technological approach to learning. However, the 
complexity of the concept of critical thinking and the assumptions about students’ technological skills are 
debatable issues that require clarification and evidence-based research in terms of teaching and learning. This 
paper reports on a case study of an online Blackboard site at the University of Western Sydney, where analysis 
of patterns of usage of the online site and qualitative analysis of student feedback provide evidence to support 
its effectiveness for encouraging students' critical thinking. There is potential to expand this into a more widely 
usable teaching and learning resource in the future, and for further research to explore the benefits for student 
learning. 
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Introduction 

Background 

Critical thinking is expected of students from the first year in higher education (HE) onwards. One 
of the most difficult challenges for students in the transition phase to academic writing in HE is 
critical thinking. Lecturers frequently comment that students are not "critical enough"; however, 
students do not always understand what this means (Price & O’Donovan 2007). A website 
dedicated to developing critical thinking is one way for students to begin the process of exploring 
the nature of critical thinking in relation to their discipline area. While the literature on critical 
thinking has been historically somewhat focussed on the debate about whether critical thinking 
skills are generic (Norris & Ennis 1990; Swartz & Perkins 1990) or subject-specific (McPeck 
1990; Moore 2004), in more recent years there has been a definite move towards a focus on 
learning and teaching. This is emphasised by those proposing a constructivist approach to critical 
thinking (Laurillard 2002; Moon 2008). This paper is more concerned with contributing to 
information regarding students’ development of critical-thinking skills, particularly in relation to 
an online learning context, than to the debate about the essence of critical thinking. 

The use of online learning resources can be of assistance in the process of continually developing 
students’ critical skills - for “Gen Y” students as well as for other cohorts within the university 
community. Moreover, it has become an internationally accepted practice for universities to use 
electronic programs to enhance student learning, including many "new technologies" or tools that 
may have had an interactive component, such as CD-ROMs, the Internet, video and online 
databases (Barron & Orwig 1997). Current technological learning tools include digital video and 
audio equipment (notably, iPods); learning-management systems such as Blackboard; RSS feeds 
and blogs; and even social communication sites such as Wikis, Facebook, Twitter and YouTube. 
Interestingly, debate is also occurring regarding the technological competence of students. Prensky 
(2001; 2005; 2007) argues that technologically adept students need to learn in a technologically 
enhanced environment. However, research in Australia (Kennedy, Dalgarno, Bennett, Judd, Gray 
& Chang 2008) warns that presumptions of students’ technological competence may seriously 
hamper student learning. 

In 2006 a Blackboard site was set up at the University of Western Sydney (UWS) to address the 
expectation that students need to acquire critical thinking as part of their professional and lifelong 
learning skills. While many commencing students at UWS are high-school leavers, and are 
commonly assumed to be quite comfortable with the use of information, a number of these and 
other new students are challenged by the academic culture they encounter. This is to be expected, 
as over 50% of students are the first in their families to attend university; approximately 30% are 
mature-age students; and approximately 30% are from Culturally and Linguistically Diverse 
(CALD) backgrounds, both local and international (Office of Planning & Quality 2011; Reid 
2003). This paper will give a brief overview of the literature about critical thinking and online 
learning before examining a case study of the critical-thinking site at UWS. 

The nature of critical thinking 

Because "critical thinking" is an umbrella term used for a complex set of thinking skills, it is 
important that the term be clarified for HE students. There are many definitions of critical 
thinking; this can be confusing for students unless each subject lecturer is explicit about their own 
meaning (Lloyd & Bahr 2010; Moon 2008), particularly if marking criteria include critical 
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analysis. A working definition of critical thinking that seems transportable to most HE contexts is 
Moon’s (2008, p7): "Critical thinking is a capacity to work with complex ideas whereby a person 
can make effective provision of evidence to justify a reasonable judgement. The evidence, and 
therefore the judgement, will pay appropriate attention to context". Furthermore, Duron, Limbach 
and Waugh (2006) emphasise the higher levels of thinking involved, such as evaluation and 
reflection. For students, these descriptions highlight that critical thinking is a sophisticated skill set 
rather than a superficial requirement easily acquired by rote learning. 

The process of clarification of the term "critical thinking" may serve, in fact, to highlight the 
complexity of the term and the various skills it encompasses, such as analysis, making judgements, 
problem-solving, evaluating, questioning, reflecting and the like. Such expected skills may even 
vary between different units of study within a discipline (Lloyd & Bahr 2010). At the same time, it 
has to be taken into consideration that students are at different developmental levels of thinking 
(Perry 1970). More recently, an extensive study of undergraduate and post-college students 
(Baxter Magolda 1992; 2001) identified four levels of thinking. The basic level, absolute knowing, 
describes students who hold a black-and-white view of the world. At level two, transitional 
knowing, students accept that uncertainty is a reality in some areas of knowledge; and at level 
three, independent knowing describes students who can come to a personal conclusion. The 
highest level, contextual knowing, is achieved when students appreciate that a particular 
perspective may be affected by its context. 

Critical thinking and technology 

Bringing together critical thinking and digital technologies may be beneficial in that it provides an 
additional opportunity for interested students to achieve Baxter Magolda’s (1992; 2001) higher 
levels of knowing and/or to practise the seven processes of critical thinking (Carmichael, Craigie, 
Driscoll, Farrell, James & Scoufis 1998). Since the Internet is an increasing source of information 
for learning (Astleitner 2002), it is also important for students to develop critical-thinking skills to 
apply to web-based materials. 

Moreover, there is global recognition of the importance of information literacy for a variety of 
everyday tasks, for lifelong learning and for successful engagement with the local and 
international community (Lorenzo & Dziuban 2006; UNSW 2011). The term "information 
literacy" can be defined as: "the ability to locate, evaluate and use effectively...needed 
information" (Webber, Boon & Johnston 2005). Prensky (2001) and Oblinger and Oblinger 
(2005), writing from the North American perspective, support the notion of current university 
students' competence with a range of personal and social technologies such as iPods, the advanced 
features on mobile phones and social networking sites. Prensky (2001) in particular claims that 
current HE students are greatly disadvantaged by teaching that does not take into consideration 
their lived experience. 

Yet, while acknowledging the social technology skills of Australian HE students, Bennett, Maton 
& Kervin (2008) question their overall technological skills and information-literacy levels in terms 
of approaching academic tasks. In Australia, recent research has reported on survey findings from 
three universities where 108 university staff and 2,588 students were surveyed regarding their use 
of technology (Kennedy et al. 2008). Surprisingly, one finding was that many first-year students 
were "relatively unfamiliar with a range of emerging technologies and tools" (p. 490). Also, 
Kennedy et al. (2008) found that cultural and socio-economic factors seem to affect the usage 
patterns of technologies among some students (Bennett et al. 2008). The implication of these 
findings is that HE providers cannot assume sophisticated technological skills among their 
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students, and need to take a variety of factors into account when designing web-based materials. 
Moreover, while graduates typically display high levels of proficiency in using the required 
technology once they are in the workplace, they still lack critical-thinking skills (Lorenzo & 
Dziuban 2006). Thus care needs to be taken when designing learning materials that the technology 
is appropriate for the students who will use it, so that they can understand and, in this case, learn 
about critical thinking. 

Learning online in HE - considerations for designers 

As with face-to-face situations, successful online learning depends on high-quality planning and 
delivery, appropriate learning tasks and timely feedback (Clark & Mayer 2008). In the last decade 
or more, researchers have been attempting to develop a composite picture of the effectiveness of 
online learning (Astleitner 2002; Perez 2002; Bravo, Van Joolingen & De Jong 2006; Coates 
2007; UNSW 2011). Learning-management systems (LMSes), such as Blackboard, are commonly 
used in Australia and elsewhere as a part of the student learning experience (Coates 2007). 
However, little is known about the impact of LMSes on student learning and learning outcomes 
(Coates 2007). For example, Coates surveyed over 1,000 students across four disciplinary areas at 
four institutions to gauge the level of academic and social engagement of commencing students 
who used online LMSes. From the data obtained from the study, Coates developed a typology 
suggesting four likely types of online engagement: "intense, collaborative, independent or passive" 
(pi32). Briefly, intense engagement correlates with high levels of involvement with study, 
students and staff; collaboratively engaged students prefer the social aspects of learning over 
independent learning; an independent style of engagement is characterised as highly motivated 
academically but less socially; while passive engagement is used to describe those students whose 
participation in all university activities is low. This research goes some way toward an 
understanding of how students engage with online learning, and how to identify student 
competence with the technology in an Australian university context. 

Taking a pragmatic approach that is localised to the institutional context, Clark and Mayer (2008) 
suggest that the beginning point for the development of effective web-based resources is 
consideration of how the e-Learning architecture will reflect different learning purposes. These 
can be described as: information acquisition; a directive approach or "guided discovery" (p27) 
through the use of interactive activities; and student-created educational resources that are shared 
beyond their own institution. Guided discovery supports users’ construction of knowledge and 
their ability to apply this new knowledge to other contexts, and is therefore the architecture most 
suited to the development of critical thinking. Ideally, Keats and Schmidt (2007) argue, the social 
interactivity of technology - especially in relation to sharing and discussion of ideas - has the 
potential to connect HE students to the international community to create a wider socio-political 
learning environment. This wider environment should provide additional opportunities to develop 
critical thinking. This categorisation is considered to be appropriate for the case study to be 
examined in this paper. 

Beneath the e-learning architecture sit four key pedagogical considerations: content, instructional 
methods, practice examples and feedback (Clark & Mayer 2008). These considerations are 
important, since together they can support quality student learning, provided students are 
technologically proficient and have easy access to learning tools and resources. Good educational 
design also entails a supportive physical and social environment (Ellsworth 1994). The physical 
environment includes IT resources that are available to students off campus and the university’s IT 
support structures, as well the design of the materials themselves (Goodyear 2005). These need to 
be compatible to avoid student dissatisfaction. The social environment will usually involve social 
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networking with peers and facilitators on open-forum discussion boards, and may include private 
feedback from the facilitator (Goodyear 2005). 

Critical thinking skills needed for professional and lifelong learning are also important attributes 
for technology-based learning environments, so that learners can synthesise and evaluate sources 
of knowledge and integrate learning from social-networking sites. To maximise learning, care 
needs to be taken to engage HE students who may vary in their learning development, interests 
and experience with technological skills. However, the blending of technological tools with the 
development of critical-thinking resources offers a potential for HE students to develop lifelong 
learning practices and vital social networking; hence the current case study. 

The case study 

The critical-thinking Blackboard site was established to provide an interactive resource for 
independent and group student development of the concept and practice of critical thinking. One 
approach to address the variation in students’ developmental levels, cultural backgrounds and 
academic experience is to draw on terms reported by HE students themselves to describe critical 
thinking. These terms can then be used as a starting point to explore more commonly used abstract 
terms when discussing critical thinking. This approach was first used in a book of sample essays 
that successfully demonstrated critical thinking in six undergraduate disciplines (Carmichael et al. 
1998). The researchers interviewed students who had been awarded a credit grade or higher for an 
assignment that the marker identified as exhibiting critical thinking. The interviews explored the 
writers’ and markers’ perceptions of critical thinking and the textual features in these assignments 
that expressed it. These observations were used to annotate the sample assignments. The collated 
data from these student interviews suggested seven processes that the student writers frequently 
used to achieve a critical approach in their writing. These processes are iterative and interrelated, 
and the learning modules on the Blackboard site at UWS represent each process (Figure 1). This 
provides an opportunity for students to learn about different aspects of critical thinking. While this 
is not explicitly teaching critical thinking (Choy & Cheah 2009), it is a resource for students to 
explore the processes that other students have used in developing their critical thinking (Farrell, 
James, Carmichael & Scoufis 1997). 
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Figure 1: Critical thinking site: learning modules 


Table 1 names each process, defines its meaning and provides examples of exercises to be found 
in that particular module on the site. Figure 2 illustrates the introductory module: What is critical 
thinking? As well as some text and short video clips, there are links to the seven processes that 
were identified by students in the previous study. 


Table 1: Explanation of critical thinking processes in the learning modules 


Process 

Descriptor 

Example of an exercise 

1. Question 
interpretation 

Analysing and interpreting the 
question, before researching and 
reading the literature 

Exercise on breaking down the 
question into topic, instruction 
words and focus. Connection 
between question and point of view 
expressed in example. 

2. Immersion 

Understanding the topic through 
reading, discussion and thought 

Quiz about how students immerse 
themselves in a topic; e.g., reading 
widely. 

3. Questioning 

Asking questions that help to 
explore the topic 

Developing a concept map of 
questions; e.g., Why is this issue 
important? 
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4. Making links/ 
connections 

Tying together a concept with 
similar/opposing concepts in 
another field or related to 
workplace practice or life 
experience 

Quiz relating theories to "real life" 
practical situations, for example 
work/life balance. 

5. Views 

Understanding the topic from a 
variety of perspectives, evaluation 
of different views 

Work-related problem scenario 
asking for a solution according to 
the perspective of an 
employee/manager/HR 
professional. 

6. Frameworks 

Understanding the relevant 
concepts within a wider context of 
a paradigm and/or theory within 
the subject or course 

Finding concepts, theoretical 
perspectives or paradigms in a short 
text. 

7. Position 

Developing a position and the 
arguments to support it 

Use of a concept pyramid to 
compare theories and evidence. 


Adapted from Scoufis et al. (1999). 



Figure 2: Learning module: What is critical thinking? 

Design of the online critical thinking site 

This Blackboard project is an extension of previously funded projects at UWS encouraging the 
development of students' critical thinking. The project encompassed Goodyear’s (2005, p2) 
principles, such as “designing good learning tasks, ...good access to robust and appropriate 
technology and helping create a convivial learning culture”. Blackboard was chosen as the LMS 
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because students are familiar with the system, as it is a common learning tool across the 
university; there is a high level of system compatibility both on and off campus; and Blackboard is 
supported by the university’s IT unit. The Blackboard critical-thinking site encompasses 
interactive learning modules, multi-media segments, the latest edition of the e-book containing 
sample assignments (Carmichael et al. 2009) and an opportunity for students to communicate via 
email and discussion board with each other and the site instructors. A place for a reflective journal 
entry is supplied at the conclusion of each learning module. The design of the site is based on a 
clear understanding of its purpose (Perez 2002; UNSW 2011). The site is open to all students at 
UWS, and is also available for lecturers to use in class to stimulate discussion about critical 
thinking. 

The pilot online project began in 2006, when materials were trialled for (i) a site that was available 
to all students at UWS and accessed through the Student Learning Unit (SLU) portal; and (ii) a 
module added to the Blackboard site for the subject Engineering and Industrial Design Practice 
(EIDP). In 2009 the SLU online resource was further developed, the e-book was updated with 
additional model assignments from the fields of arts, social sciences and environmental studies, 
and a specific cohort of students (those enrolled in the subject Business Academic Skills) made 
use of the site through a more embedded approach. This initiative had instructional input as well as 
incorporating the independent, social and collaborative aspects of online learning (Astleitner 
2002), and obtained evaluative feedback from students to further develop the resource. 

While there is a strong element of a student-centred constructivist approach (Laurillard 2002) in 
this site, it also embodies a "free access" approach (Perez 2002, p225) that allows students to 
explore materials according to their needs. The learning modules were developed from the 
interviews with students and lecturers, with explanations of critical thinking by lecturers and 
students, in video and audio formats. Other modules are based on the seven processes identified 
above (Table 1). Interactive exercises and quizzes are included in most of the learning modules. A 
preliminary quiz directs students to the appropriate module/s to enhance their understanding of 
critical thinking. Another quiz asks them to identify strategies that would help develop critical 
thinking, or identify issues from various scenarios. Students are also provided with a description of 
the processes and an example, often taken from student writing. However, what evidence of 
student learning can be gleaned from their use of this critical-thinking site? 

Within this context a research study was undertaken to evaluate this online critical-thinking 
resource at UWS. An important purpose of the research was to ascertain whether students gain an 
understanding of critical thinking and are able to transfer this to their discipline and new contexts. 
The project not only aimed to stimulate students' critical thinking and independent learning, but 
also to encourage discussion within the academic community, which had occurred at conference 
presentations in 2009. 

Methodology 

Recent research has looked at trends of student usage of online resources (Coates 2007; Kennedy 
et al. 2008). This evaluative study will add a more in-depth view to the composite picture to date 
in attempting to examine both the student interactions with the materials and their perceptions of 
the benefits of using the site. An integrated methodology based on pragmatism and using 
numerical and qualitative analysis (Denzin & Lincoln 2003) provided information about usage 
patterns and students’ learning experiences. As Creswell (2003, pl2) indicates, a pragmatic 
approach can use mixed methods that are relevant to finding out “knowledge about the problem”; 
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this was the approach used in this case study. First, a numerical analysis of student usage patterns 
of the two pilot sites established in 2006, SLU and EIDP, was undertaken. In 2009 a similar 
analysis was carried out. Analysis of student usage of the site in the first semester of 2011 was also 
determined. In both 2006 and 2009 a questionnaire was distributed, followed by semi-structured 
telephone interviews, and each was analysed thematically as qualitative data. 

Findings and Implications 

Patterns of usage - what evidence do they provide? 

A total of 45 students logged on to the generic SLU website in May 2006, the first month of its 
operation. Various patterns of usage emerged; for example, 33 students had fewer than 10 visits to 
the site, six students had between 11 and 20 visits and six accessed the site more than 21 times 
(Figure 1). The highest users (32-105 visits) were from a variety of faculties (Law, Nursing, Arts, 
Adult Education). It could be expected that they comprehensively explored the site; for instance, 
one student accessed the site 93 times to view 90% of the site. This student's last visit was to 
explore the pages that describe critical thinking. A second student accessed the site 87 times to 
view 84% of the site. This student completed an investigation of the site by looking at Handy 
Hints, a page that provides some critical-thinking strategies. In contrast, a mid-range user (14 
visits) viewed five pages of the site repeatedly and ended usage of the site by exploring 
descriptions of critical thinking. Unfortunately, due to the under-development of university-wide 
Blackboard reporting practices for the first semester of 2006, it is not possible to ascertain how 
much time each student spent on the website. 



Figure 3: Range of student visits to the SLU site in May 2006 (N=45) 

However, by the end of 2006 reporting practices had been expanded, and it could be determined 
that a total number of 66 students had made 1,448 voluntary visits to the site. Length of time for 
each visit varied from 18 seconds to more than three hours, and the number of visits ranged from 
one to 27. In general, the fewer the visits to the site, the more time was spent, but there were some 
notable exceptions. For instance, one student had seven visits for a total of 12 minutes; another 
accessed the site five times for a total of three hours and explored sample essays and modules on 
the aspects of critical thinking; one student’s only visit lasted 2 hours 51 minutes. Again, a range 
of disciplines was represented. 

Comparison of the usage patterns of the SLU critical-thinking website with the discipline-specific 
EIDP Critical Thinking module was undertaken. The latter site used the same learning modules, 
but only supplied the engineering model assignments. Approximately 300 students were enrolled 
in this first-semester core unit, which takes an enabling approach to inducting students into their 
discipline. The EIDP site was used to instruct, support student learning and serve as a 
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communication tool. A random sample of the use of the critical-thinking modules by the 300 
students was based on every 20 th user (N=14). Usage of the entire EIDP site was spread over the 
semester, and ranged from 206 to 2,890 visits in total. Visits to the critical-thinking folder within 
that site were clustered in early May (13 of 14 students), with this sample of students mostly 
visiting the critical-thinking modules over one to two days, although several had visits spread over 
one to two months. Students had been advised to use the site over a two-week period as part of 
their course. It was not possible to determine exactly how many students accessed the critical- 
thinking materials on the EIDP site. Table 2 shows how this small sample of students made use of 
the materials. 

Table 2: Variation in patterns of usage for random sample of students EIDP (N=14) 


Modules in the folder 

Number of student visits 

Number of repeat visits 

Descriptions of critical 

13 

4(1 returned 6 times; another 

thinking 


5 times) 

CT slide show 

4 

1 (returned twice) 

Handy hints 

11 

1 (student returned 3 times) 

Seven processes of CT 

1 

0 

Self-diagnosis of 
understanding of CT 

1 

0 

Sample texts demonstrating 

CT 

8 

3(1 returned 4 times) 

Practice exercises 

8 

4 (1 student returned 3 times; 
another 2) 

Answers to exercises 

3 

0 

Introduction to model 
assignments 

7 

3 


4 

1 (returned 4 times) 

Model Assignment 1 

6 

0 

Model Assignment 2 

Model Assignment 3 

5 

2 


This snapshot of usage patterns indicates that students’ exploration of the site varied considerably. 
For instance, students 4 and 9 seemed to take a somewhat logical but "short cut" approach, 
beginning with a brief overview of critical thinking, handy hints, a practice example and answer 
and finally another practice text and answer. Student 4 then accessed the model assignments in 
order on three different days, suggesting a need to gain more in-depth knowledge. Students 8,11 
and 12 all took a methodical approach by accessing the conceptual items (definitions, descriptions 
and slide show) followed by examples and practice exercises. However, only Student 12 of this 
sub-group looked at the suggested answers. Two looked at the introduction to model assignments 
but did not explore the models. In terms of the range of approaches, the students using either site 
demonstrated a "free access" approach (Perez 2002) or independent learning, following their 
interests or taking a "need to know" approach. Returning to particular items seems to suggest 
engagement (Coates 2007). Closer analysis of the EIDP site demonstrates in more detail the 
variation in student usage found through the SLU site. 

Usage patterns in 2009 show some differences to those in 2006 (Table 3). Student use of the site 
was voluntary, though one group of students in a first-year core subject, Business Academic Skills 
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(BAS), was encouraged to use the site as part of a learning portfolio. This group presumably 
contributed to a higher usage rate in 2009, when 904 users were recorded. These users made 2,821 
visits to the site, ranging from one to 66 return visits for any one individual. While the number of 
users of the SLU site has greatly increased, the time spent on the site for many students has been 
minimal. For instance, in 2009 44% of the users spent less than 10 minutes on the site. The 
average user time was 41 minutes and the most time spent by any user was 12 hours. However, 
time spent on a site may be complicated by students leaving the computer on while they do 
something else, or they may print out information to read later. The usefulness of the site for some 
students can be measured by those who revisited the site over a nine-month period. An 
examination of several students’ use of the learning modules in 2009 indicated that, as noted 
above, they tended to "dip" into modules randomly, as it suited them (Perez 2002), and spent 
differing amounts of time exploring any one module. Students were interested in the discussion 
messages, with 30% of them reading the messages regularly. However, only 7% actually 
contributed to the discussion by posting a message. Twenty-seven percent attempted the quizzes, 
while most students viewed the learning module and book files between one and 127 times. Late at 
night was the most common time students used the site. This pattern of usage continued to 
demonstrate a "need to know" approach. Those who could be seen to "overuse" the site may have 
been genuinely learning from revisiting modules or model assignments, or were perhaps seeking 
more interactive communication with peers (Coates 2007). 

Finally, in the first half of 2011 there have been 1,170 visits to the site from 520 users (Table 3). 
The maximum time spent on the site by any one user was 9 hours and 35 minutes, with a 
maximum of 20 return visits by any one student. Sixty students spent over half an hour using the 
site, with 39 of those using the site between one and 9.5 hours. Again, it is uncertain whether 
extended time spent on a site may be complicated by students leaving the computer unattended, or 
whether short logon times are indicative of printing out information to read in hard copy. 


Table 3: Variation in patterns of usage for students using the site, 2006-2011 


Site 

Number of users 

Number of visits 

Time accessed 

Number of 
return visits 

2006 SLU site 

66 

1,448 

18 seconds - 3 
hours 

1 -27 

2006 EIDP site 

300 

206 - 2,890 
(entire EIDP site, 
numbers unknown 
for critical 
thinking site) 

2 days - over 2 
months 

Different 
patterns (see 
samples in Table 
2) 

2009 SLU site 

904 

2,821 

22 seconds - 12 
hours 

1-66 return 
visits 

2011 SLU site 
(semester 1) 

520 

1,170 

4 seconds - 9.5 
hours 

1-20 return 
visits 


Student feedback on the usefulness of the site 

To gain student feedback, a questionnaire was distributed to 448 students who had used the site 
during 2009, with follow-up telephone interviews. Responses from 113 students (25%), the 
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majority of whom (64%) were studying business, were obtained. This may relate to the learning 
portfolio for BAS students, or to the fact that the largest cohort of students at UWS is business 
students. The remainder were from various degree courses including nursing and health-related 
disciplines, teaching/education, sciences and arts. Year of study varied: approximately 68% were 
first-year students, 27% were second- or third-year students and 5% were fourth-year or more, 
including postgraduate and part-time students. 

Numerical analysis of the questionnaire responses was indicative of student engagement with the 
critical-thinking process as a result of using the site (Table 4). Importantly, 95% of the respondents 
would recommend this site to other students, of whom 73% indicated that they found the site to be 
useful by rating it 3.5 or above on a Likert scale of 1 to 5, where 1 = not useful and 5 = very 
useful. Although only 38% of the students responding to the questionnaire had read sample 
assignments in the e-book, all who did read them indicated that they were useful. This deeper 
reading of information is more suggestive of Coates's (2007) notion of intensive and independent 
engagement. On the other hand, 67% of the students found the exercises and activities in the 
interactive learning modules to be useful, indicating that more students prefer the instant feedback 
provided in those modules. Ninety-four percent of respondents did not find the modules difficult to 
understand. 


Table 4: Responses to feedback survey (N=113) 


Question 

Yes 

N/A or No 

Total 

Would you recommend 
this site to other students? 

95% 

5% 

100% 

Was the site useful in 
terms of understanding 
critical thinking? 

73% 

27% 

100% 

Did you read the sample 
student assignments in the 
e-book? 

38% 

62% 

100% 

Did you find these sample 
assignments useful? (N= 
33) 

100% 

0% 

100% 

Did you find the 
interactive exercises in 
the modules useful? 

67% 

33% 

100% 

Did you find the modules 
too difficult to 
understand? 

6% 

94% 

100% 


Qualitative evaluation of student experience of the site 

The numerical analysis of student usage and engagement shown in Table 4 indicates that students 
took different approaches to exploring the site and found it useful. However, this data does not 
delve into the students' experiences of their learning. Qualitative information was gained from the 
general SLU site in 2009 via questionnaires, emailed comments on the site and semi-structured 
interviews (Freebody 2003). This provided a more in-depth understanding of learning for students, 
especially undergraduate students. 
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Students who interacted with the instructor via the discussion board on the general SLU site from 
2006 initially discussed practical questions, such as access problems, but also expressed positive 
feedback that suggested they were motivated and engaged, and which corresponded to Coates’ 
intense engagement (Coates 2007). Statements included: "I’ve found the site helpful and hope I 
can now put it into practice" and "this site is really great, just wish I looked at it sooner". Others 
were enthusiastic, with statements such as: "I wanted to comment on how important and helpful 
this site has been...[I] have an exam where critical thinking will be questioned and the 
information found here has really helped". More recently, students have interacted with their peers 
in comments such as: "I agree with XXX, our life experiences assist us with analysing the given 
information...." 

Information relating to their perceptions of the usefulness of the site was obtained through student 
comments in response to the questionnaire. One theme that emerged was that the information and 
exercises were helpful for writing assignments, and with various subjects. One student said ‘[I] 
liked doing it at home in my own time. It helped for questioning information when researching to 
make up a table with both sides of the argument’. Other students appeared to believe that the 
modules had scaffolded their learning about academic writing, such as the following comment: 

‘Helped the transition between high school and uni. ’ An undergraduate business student made the 
following observation about writing an essay for Macroeconomics: ‘The part about reading 
different texts and contrasting those ideas with the assigned text book was particularly helpful... 
The site provided a good outline to start off an essay ’. This indicates a broader context in which 
students could synthesise information and transfer it to other situations (Lorenzo & Dziuban 
2006). This reported transference in three contexts by three separate students, is indicative of 
respondents’ confidence and ability to operate at Baxter Magolda’s higher levels of thinking: 
Independent Knowing or even Contextual Knowing (1992; 2001). 

Some students made comments specifically about the accessibility of the content on the site and 
how it helped their understanding of critical thinking. For example, one student stated that 
‘examples were easy to understand, helped define what critical thinking is. Quizzes were useful'. 
Another student explained that the site ‘helped me to understand critical thinking and pinpoint 
areas to analyse text [to] use in my assignment’. One international Nursing student made the 
following comments on her questionnaire: ‘it was very useful as I am a Mature Age student and 
from Brazil. It gave me ideas about the way to study in Australia. I read through it from beginning 
to end and I answered all the questions. Exercises were very useful as I got knowledge and 
feedback’. Another student said: ‘The examples and descriptions are not too short, but to the 
point. You don’t have to read too much ’. This is reinforcing of the approach taken in developing 
the interactive online materials as short, bite sized sections for ease of reading (Perez 2002), and in 
keeping with the principles of educational technology espoused by Goodyear (2005) and Clark and 
Mayer (2008). One student found the site ‘ unhelpful ’ as she wanted to gain high distinctions for 
her essays. After several email communications with the site instructor, she began sharing 
information and resources regarding academic argument with the other students via the Discussion 
Board. This demonstrated a change in attitude and a more mature interaction with the site, 
particularly when she began to achieve HDs for her assignments. 

Of the seven learning modules, the students rated the most useful modules to be Questioning and 
Question Interpretation, which are possibly the most "concrete" of the modules, and a practical 
starting point for critical thinking. These preferences were supported by a closer examination of 
the learning reflections written by 50 students, which indicated that the former module had raised 
awareness of breaking the question into key parts: topic, instruction words and focus. One student 
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wrote: "before I have never concentrated too much on analysis [of the question], too time 
consuming, especially in exams [now] I realise it is not a waste of time". Another wrote that "this 
activity is really very useful, it helped me to understand the first aspect of thinking critically, that 
is to understand the question". Many of the students’ reflections indicated that interpreting the 
question was important for guiding a clearer and better essay answer. This is evidence of 
stimulation of student thinking, albeit at a rudimentary level. 

Furthermore, the reflections suggest that students learned useful strategies from the modules, 
which assisted their approach to critical thinking in assignment preparation. For example, one 
student wrote: "this module helped me to understand that ...I have to take into account the 
different views that are within the text"; another student commented about developing a 
questioning approach: "now I am better equipped to put into practice my understanding..." One 
student explained, "I did not realise how important questioning is until I read through this 
topic...the concept map was also very helpful as it addressed all the issues we should explore 
before we answer a question ". 

Another theme in the questionnaire comments was that the assignments in the e-book were good 
"example assignments/papers" that gave them more confidence in understanding what was 
expected. This was acknowledged by a number of students, with one stating that it gave "insight 
into how to answer the question". Another stated that the most useful aspect of the model 
assignments was "teacher and student comments". Overall, these reflective comments suggest that 
the respondents gained insight into the complex term "critical thinking" in relation to their 
academic writing. 

One BAS tutor, who recommended the resources to her class during 2009, also indicated that 
student learning was enhanced by using them. She taught approximately 100 students altogether in 
four separate tutorial groups. Over 95% of the students were first-years. The site, she suggested, 
provided transitional support for new students: "for a lot of students seeing what actually qualifies 
as a good piece of writing is really very important because coming in from high school they 
haven’t seen the level of thinking that is required". After marking assignments, she reviewed 
student usage patterns of the site and identified that those students who had used the site achieved 
better marks in their reflective assessment than those who had not used it. 

Conclusion 

The literature indicates that the success of using online resources for the development of students' 
critical thinking in the HE context at least partially depends on students’ developmental levels, 
their experience with the technology used in academic settings and their levels of engagement. 
Successful learning about critical thinking in an online environment is also contingent on the 
quality of online resources and tools. This case study demonstrates that many students find online 
learning about critical thinking to be helpful, stimulating and engaging. It verifies that some 
students enjoy learning in their own space and time and that this site contained suitable content, 
sample texts, practice examples and timely feedback (Clark & Mayer 2008). 

Research suggests that a site embedded within a unit of study with instructional and peer 
interaction could achieve better critical-thinking outcomes (Ramsden 1992; Skillen 2006) in 
relation to teaching skills, whether they be thinking or writing skills. However, findings in this 
case study indicate that stand-alone resources can achieve perceived benefits for students. Further 
research comparing student understanding of critical thinking via a stand-alone site with one 
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embedded in a face-to-face learning environment would offer an opportunity to more closely 
evaluate student outcomes. This could include indicators of learning in terms of assignment marks 
or grades. 

There is also potential to expand the current site into a freely accessible website, with opportunity 
for greater national and international interaction together with interesting research opportunities 
(Keats & Schmidt 2007). This could provide students with the experience of technology to 
broaden their perspectives regarding critical thinking through networking in a wider cross- 
institutional and cross-country environment. 
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